The quality of systematic reviews/meta‐analyses assessing the effects of ankle–foot orthosis on clinical outcomes in stroke patients: A methodological systematic review

Abstract Background and Aims Given the importance of systematic reviews (SRs) for practitioners, researchers, and policymakers, it is essential to assess them to ensure robust methodology and reliable results before applying them. The purpose of this methodological study was to assess the methodological and reporting quality of recently published SRs and/or meta‐analyses (MAs) evaluating the effects of ankle–foot orthoses (AFOs) on clinical outcomes in stroke survivors. Methods PubMed, Scopus, Web of Science, Embase, ProQuest, CENTRAL, REHABDATA, and PEDro were searched. The research team applied A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR‐2) tool and Preferred Reporting Items for Systematic Reviews and Meta‐analyses (PRISMA) checklist for evaluating the reporting and methodological quality, respectively, and the ROBIS tool was used to evaluate the risk of bias (RoB) in the included reviews. The quality of the evidence was also judged using the (Grades of Recommendation, Assessment, Development and Evaluation) GRADE method. Results In final, 14 SRs/MAs met inclusion criteria. Evaluation of methodological quality using the AMSTAR‐2 tool demonstrated that the overall quality of included reviews was mostly “critically low” or “low,” except for two studies that were “high.” In addition, the findings showed that the mean score of the reporting quality of the included reviews based on the PRISMA criteria was 24.9, down from 42. In accordance with the overall evaluation applying the ROBIS tool, 14.3% of the review studies were evaluated as high RoB, 64.3% were evaluated as unclear RoB, and 21.4% were evaluated as low RoB. Regarding the level of evidence quality, the GRADE results indicated that the evidence quality of the included reviews was unsatisfactory. Conclusion This study showed that although the reporting quality of recently published SR/MAs evaluating the clinical effects of AFOs in stroke survivors was moderate, the methodological quality of almost all reviews was suboptimal. Therefore, reviewers must consider a number of criteria in designing, conducting, and reporting their studies to move toward transparent and conclusive results.


| INTRODUCTION
Stroke is a major cause of death and disability in the world, with a high rate of chronic disability in survivors. 1 Over the past three decades, the annual number of strokes has increased substantially, and its burden is likely to increase globally, particularly in low-income countries. 2 Rehabilitation measures are taken to improve the functional outcomes of stroke survivors. An orthosis is defined as a device aimed at correcting and supporting the structure and function of the musculoskeletal system. 3 Ankle-foot orthoses (AFOs) are among the most commonly used rehabilitation options. AFOs are used to stabilize the foot and ankle during the stance phase, keeping the toes up while walking, and allowing for heel strike. Thus, AFOs are used as a rehabilitation intervention to improve the gait of stroke patients. 4 Until now, many trials have studied the effects of AFOs on clinical outcomes in stroke survivors, and their data have been analyzed by relevant systematic reviews (SRs) and meta-analyses (MAs). SRs are considered the gold standard for evidence used to assess the effectiveness of an intervention, 5 and high-quality SRs play an important role in evidence-based clinical decision-making for practitioners. Since some published SRs and MAs are not of high quality, precautions should be taken when using their findings as valid results. In fact, it has been stated that the large majority of SRs and MAs in biomedicine are likely redundant, misleading, and/or conflicted. 6 Given the importance of SRs and MAs for practitioners, researchers, and policymakers, it is essential to assess them to ensure robust methodology and reliable results before applying them. In fact, some bias caused by a range of factors (e.g., reporting and methodological) can affect the validity and reliability of findings from SRs/MAs. 7 Currently, no methodological study has evaluated the quality of SRs assessing the effects of AFOs and identified the main factors affecting the quality of such studies. Various tools and checklists have been developed in recent years to evaluate various aspects of these studies. This methodological SR is the first study to appraise the methodological and reporting quality of the recently published SR/MAs assessing the effects of AFOs on the clinical outcomes of stroke survivors.

| METHODS
This study has been written in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) criteria. 8 Furthermore, the research team considered the criteria of A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR-2) to enhance the reporting quality. 9 The protocol of this study was already reviewed and registered with the Iran National Committee for Biomedical Research (Registration number: IR.SUMS.-REC.1401.241, available here: https://b2n.ir/s72067).

| Search strategy
Three components of PICO-S (population, intervention, comparison, outcome, and study design), including population (stroke patients), intervention (AFO), and study design (SR/MA), were adhered to when developing search strings. The research team tried to identify the most relevant terms using Medical Subject Headings (MeSH) and Emtree thesauruses. In addition, a free-text approach and contacting relevant experts were used to find more terms. The search string was first created for the PubMed database and then adapted for other electronic databases. Keywords such as "ankle-foot orthosis," "ankle foot orthosis," "ankle-foot orthoses," "ankle foot orthoses," "Foot orthotic devices," "stroke, strokes," "Cerebrovascular Accident," "Cerebrovascular Accidents," "CVA," "Cerebrovascular Apoplexy," "systematic review," "review," "meta analysis," "metaanalysis," and "meta analyses" were considered to develop search strings (Supporting Information: Table 1) All of the initial records identified through searching electronic databases were imported into Endnote X8 software (Thomson Reuters) to withdraw the duplicates, and then the remaining records were screened based on the title and abstract by two reviewers (S. Sh. and M. J.) independently. In the next steps, the same two reviewers independently began assessing the full text of potential studies against inclusion and exclusion criteria to retrieve the final studies. Any conflict between two reviewers was resolved by discussion, and if required, the assistance of the third reviewer (P. M.) was used.

| Eligibility criteria
The current study included SRs and MAs who were evaluating the effects of AFO on clinical outcomes in stroke patients. The clinical outcomes of interest included gait parameters (walking speed, cadence, step length, stride length), balance, the Timed Up and Go test, the Functional Ambulation Category, energy expenditure, and hip/ankle kinematics. The eligible SR/MAs had at least evaluated the effects of AFO as one of the main primary aims in the following comparisons: (1) AFO versus no intervention; (2) AFO versus another intervention; (3) one type of AFO versus another type of AFO; (4) AFO combined with another intervention versus another intervention or only AFO. We excluded other types of review studies (including scoping reviews, critical reviews, umbrella reviews, etc.).
Protocol studies, conference abstracts, SR/MA of animals, and studies lacking full text were also excluded.

| Data extraction
This stage was performed by two reviewers (S. Sh. and P. M.) independently. As in the previous steps, any conflicts were resolved through discussion and the assistance of the third reviewer (P. M.).

| Reporting and methodological quality assessment
The research team applied the PRISMA checklist and AMSTAR-2 tool for evaluating the reporting and methodological quality, respectively. PRISMA, as a reporting guideline, consists of 7 sections (title, abstract, introduction, methods, results, discussion, and other information) and 27 items. Each item was assessed as "yes" (full reports), "partial yes" (partial reports), or "no" (no reports). 8 The adherence of each item is revealed as a ratio. The AMSTAR-2 tool, which was published recently, consists of 16 items, and each item could be judged as "yes," "partial yes," or "no." The final rating of quality can be presented as "high," "moderate," "low," and "critically low." 9 There are seven critical domains (items 2, 4, 7, 9, 11, 13, and 15) in the AMSTAR-2 tool that affect the overall rating. The research team rated the overall methodological quality in accordance with the AMSAR 2 tool as follows: (1) "high": if there is not any major concern regarding the critical domains but up to three flaws identified regarding other domains; (2) moderate: if there are more than three flaws identified regarding noncritical domains; (3) low: if there is a major flaw regarding a critical domain; and (4) critically low: if there are more than one major flaw in critical domains. 9,10 Forest plots were used to indicate the proportions of included studies meeting complete adherence to PRISMA and AMSTAR-2 criteria. To establish such plots, two-sided 95% confidence intervals (CIs) of percentages were considered using the Wilson Score method.
Statistical analyses were conducted via Stata 14.0 software (Stata-Corp LP).

| RoB assessment
ROBIS is a validated tool to evaluate the RoB within SR studies. This tool consists of three phases: (a) assessing relevance; (b) identifying concerns with the review process; and (c) judging RoB. The first phase is optional, and we did not consider it within the present study. Based on this tool, the overall level of RoB can be rated as "low," "high," or "unclear." 10,11 In final, the Grading of Recommendations Assessment, Development, and Evaluation 12 system was used to assess the quality of the evidence. Six criteria, including RoB, publication bias, imprecision, indirectness, effect size, and inconsistency, were applied to determine the level of quality as high, moderate, low, and very low. 12

| RESULTS
After the initial search, 1399 studies were found, and after removing duplicates, 265 remained, which were screened by two reviewers independently based on the title and abstract. In the next step, 37 potentially relevant studies were then identified and evaluated by the two reviewers independently based on the full text. Finally, 14 studies 4,14-26 were selected by the research team to enter the study ( Figure 1). In the final step, 23 studies were excluded, along with the reasons for exclusion, which are listed in Supporting Information: Table 2. 3.1 | Characteristics of included reviews Table 1 summarizes the characteristics of the final included reviews. The 14 review studies consisted of 8 SRs with meta-analysis, 4,14,16,18,19,23-25 4 SRs, 15,17,22,26 and 2 meta-analyses. 20,21 Based on the country of origin for the corresponding author, 37.7% of the reviews were from the United Kingdom, 28.6% were from Iran, 14.2% were from Brazil, 7.1% were from the Republic of Korea, 7.1% were from Spain, and 7.1% were from Japan.
The most common outcomes assessed in the included reviews were SHAHABI ET AL. | 3 of 20 walking speed, balance, energy consumption, gait parameters, and mobility. The Cochrane RoB Tool, the Downs and Black Checklist, and the PEDro Scale were the common RoB assessment tools that were applied in the included reviews. However, the JBI critical appraisal checklist and Jadad scale were applied in two reviews. 22,23 Notably, only two studies 18,23 stated their source of funding, and two studies 4,24 stated that they did not have financial support. As shown in Supporting Information: Figure 1, the largest number of studies were published in 2013, 2020, and 2021, with three review studies being published each year. In addition, with the exception of one study, 26 other studies have been published in peer-reviewed journals with an impact factor of 2-4.5. Table 2 shows the results of reporting quality assessment in accordance with the PRISMA criteria. As a whole, the mean adherence of included reviews to the PRISMA criteria was 24.  Evaluation of methodological quality using the AMSTAR-2 tool demonstrated that the overall quality of included reviews was mostly "critically low" or "low" (further details can be found in Table 3 and Figure 3). Interestingly, only the quality of the two studies 4,16 was high in accordance with the aforementioned tool.  Daryabor et al. 16 Daryabor et al. 17 Daryabor et al. 15 Ferreira et al. 26 Hollands et al. 23 Nascimento et al. 19 Guerra Padilla et al. 22 Prenton et al. 21 Prenton et al. 20 Shahabi et al. 4 Tyson et al.

Results
Q16a Daryabor et al. 16 Daryabor et al. 17 Daryabor et al. 15 Ferreira et al. 26 Hollands et al. 23 Nascimento et al. 19 Guerra Padilla et al. 22 Prenton et al. 21 Prenton et al. 20 Shahabi et al. 4 Tyson et al.  conducted publication bias (small study effect) assessment using a statistical method or funnel plot (item 15). Figure 4 indicates the proportions with associated 95% CIs of included reviews meeting total adherence to AMSTAR-2 items.  Table 3.

| DISCUSSION
SRs are placed at the pinnacle of the pyramid of evidence and are used to address questions and provide evidence for interventions.
This is the first methodological SR evaluating the quality of SR/MAs T A B L E 2 (Continued) Section/ topic Items Choo and Chang 18 Daryabor et al. 16 Daryabor et al. 17 Daryabor et al. 15 Ferreira et al. 26 Hollands et al. 23 Nascimento et al. 19 Guerra Padilla et al. 22 Prenton et al. 21 Prenton et al. 20 Shahabi et al. 4 Tyson et al. decades. 27 Six studies were published in the two most recent years (2020-2021), and only three studies were published in journals whose main scopes were other than rehabilitation. Walking speed was the most commonly assessed clinical outcome, presumably due to its simplicity and functional significance. 28 Six studies considered the Cochrane RoB Tool. The PEDro scale was used by three studies. This scale is an old tool, with its last modification in 1999. 29 It is primarily used for assessing the RoB in  33 This numerical rating scale was shown to fail to identify studies with increased RoB. 34 A cross-sectional analysis of F I G U R E 2 Forest plot of the proportion of included studies meeting complete adherence to PRISMA criteria. The results have been showed by ES and associated 95% CI. CI, confidence interval; ES, effect size; PRISMA, preferred reporting items for systematic reviews and meta-analyses.
T A B L E 3 Results of the AMSTAR-2 assessments. F I G U R E 3 Radar chart of AMSTAR-2 items. This chart reveals the adherence of included studies to each item of AMSTAR-2 tool. AMSTAR-2, a measurement tool to assess systematic reviews 2.
F I G U R E 4 Forest plot of the proportion of included studies meeting complete adherence to AMSTAR-2 items. The results have been showed by ES and associated 95% CI. AMSTAR-2, a measurement tool to assess systematic reviews 2; CI, confidence interval; ES, effect size.

SR protocols on health interventions registered in PROSPERO
showed that the Cochrane RoB tool was the most frequently used appraisal tool for RCTs; however, nonrandomized trials were assessed by widely varied tools. 35 In this study, the Joanna Briggs Institute critical appraisal checklist, used by one of our studies, was among the least frequently used appraisal tools. The Jadad scale, introduced in 1996, was used in one of our studies. Due to its inability to detect some serious biases, it has been recommended to discontinue its use for evaluating the quality of trials. 36 The need for a gold standard tool, especially for nonrandomized trials, is felt by authors conducting rehabilitationrelated studies. Sponsorship bias has long been recognized in clinical trials. 37  fields. 10,[38][39][40][41][42] In the current study, there was a moderate association (Speraman's ρ = 0.582, p = 0.024) between PRISMA score and AMSTAR-2 quality assessment.
SRs have been recommended to register their protocol, including major objectives, design features, and prospective analyses for the review at inception. The first advantage of protocol registration is to avoid duplications. Also, adherence to a predefined method may reduce the RoB in future results. 43  concern, which shows some eligible studies are likely to be missing from the review. Most of the reviews were graded as high concern in data collection and study appraisal domain. Errors in data collection, insufficiently available study characteristics, missed relevant results, inappropriate criteria to assess RoB, and errors in RoB assessment The results of risk of bias assessment of included studies using ROBIS tool. ROBIS is a tool to evaluate the risk of bias in systematic reviews.
T A B L E 4 Summary of findings.  may have led to this result. All of the included reviews were classified as high-risk or unclear regarding synthesis and findings. This result warns that the synthesis and findings of reviews are not free from bias. Biases may have come from different sources, like important between-study variation, inadequacies in the methodology, and incompletely reported findings.

| LIMITATIONS
The current study has a number of limitations. First, since the data extraction was unblinded, it could have been a potential source of bias in the expectations of reviewers about the quality of papers from high-ranked journals. Second, the generalizability of our findings may be restricted as papers evaluating SR/MA studies evaluate the effects of AFO on stroke patients. Indeed, AFO is an integral part of rehabilitation for patients with cerebral palsy, and there are several SR/MA studies on the effectiveness of AFO in this field. Thus, it is suggested that future studies be conducted in this field.

| CONCLUSION
This study showed that although the reporting quality of recently published SR/MAs evaluating the clinical effects of AFOs in stroke survivors was moderate, the methodological quality of almost all reviews was suboptimal because of the nonregistration of protocols, the lack of a comprehensive search strategy, the lack of a satisfactory approach to evaluating the RoB, and the failure to report the financial source of the included studies to prevent agenda bias. In addition, the quality of the evidence in many of the included reviews was low or very low. The results of the current study remind us of the necessity of critically assessing SR/MAs before using their findings and interpreting the results with great caution. It is recommended that journals ask the authors to complete checklists when submitting SR/MAs, and the reviewers also must consider a number of criteria in designing, conducting, and reporting their studies to move toward transparent and conclusive results.

CONFLICT OF INTEREST STATEMENT
The authors declare no conflict of interest.

DATA AVAILABILITY STATEMENT
The data sets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

TRANSPARENCY STATEMENT
The lead author Maryam Jalali affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.